Prediction system for texture streaming

ABSTRACT

The present invention relates to a prediction system for determining a set of subregions to be used for rendering a virtual world of a computer graphics application, said subregions belonging to streamable objects to be used for rendering said virtual world, said streamable objects each comprising a plurality of subregions. The prediction system comprisesa plurality of predictor units arranged for receiving from a computer graphics application information on the virtual world and each arranged for obtaining a predicted set of subregions for rendering a virtual world using streamable objects, each predicted set being obtained by applying a different prediction scheme,a streaming manager arranged for receiving the predicted sets of subregions, for deriving from the predicted sets a working set of subregions to be used for rendering and for outputting, based on the working set of subregions, steering instructions concerning the set of subregions to be actually used.

This application is a continuation of and claims the benefit of priorityunder 35 U.S.C. § 120 to U.S. patent application Ser. No. 15/523,446,filed on May 1, 2017, which is a U.S. National Stage Filing under 35U.S.C. 371 from International Application No. PCT/EP2015/075249, filedon Oct. 30, 2015, and published as WO 2016/071223 A1 on May 12, 2016,which claims the benefit of priority to European Application No.14196784.4, filed on Dec. 8, 2014, and to U.S. Provisional PatentApplication No. 62/074,127, filed on Nov. 3, 2014, each of which isincorporated by reference herein in its entirety.

FIELD OF THE INVENTION

The present invention is generally related to the field of computergraphics applications and in particular to applications wherein amultitude of objects are streamed for rendering a virtual world.

BACKGROUND OF THE INVENTION

To achieve visually compelling results, computer graphics applicationsmay render many virtual objects to the screen, with each virtual objectpreferably having a high level of detail. The application may typicallyhold multiple representations of those virtual objects in memory, forexample, the geometry of an object or a texture mapped on the geometry,each with a set of different levels-of-detail (LODs), such that theapplication can select the LOD representation which results in the bestvisual quality. For example, a tree at a great virtual distance does nothave to be rendered with high fidelity, so the texture for that tree mayhave a low resolution. The virtual world the application renders maycontain many more virtual objects and representations each withdifferent LODs than can fit into the application's memory.

A streaming engine is an engine that loads representations withdifferent LOD of the virtual objects in a small but fast cache memoryfrom a large but slow storage. This is done once the virtual object isneeded for processing or rendering. These representations are calledstreamable objects, for example, a texture or a geometry model. To allowfor more fine-grained streaming, the streamable objects with their LODsare split into subregions, called tiles, which the streaming enginestreams. For example, textures are split into small, rectangular regionsof pixels.

Streaming tiles from storage into memory may require a considerableamount of time. If the tile is streamed from a spinning disk to memory,the disk heads need to align, data needs to be read, tiles need to bedecompressed and copied to GPU texture memory. During this time the usermay be faced with a reduced quality of experience as the tile is notavailable for rendering. For example, the scene may be renderedpartially with no textures until the texture becomes available. Thescene may be rendered with a tile with lower quality. However, once therequested tile with higher quality becomes available, the user may bepresented with a virtual object of which the rendering fidelity suddenlychanges. In other words, the user may see additional detail ‘popping inthe screen’ once the tile is loaded.

Clearly, it is preferable to have the tiles required to render a sceneavailable into cache memory and, preferably, to have them ready in thecache memory before or at the moment the application uses them for thefirst time. If the scene can be rendered solely with tiles present inthe cache, there will be no sudden popping of additional detail in therendered results and the user can always enjoy the highest quality ofexperience.

By predicting the working set of a virtual scene, i.e., the set of tilesto be used to render the scene, the cache can be optimized and theapplication does not have to wait until the tile is available forrendering. The delay between a tile request and the moment a tilebecomes available for the application, can be hidden and the user maynever experience the aforementioned popping of detail.

Precisely predicting the working set allows for better use of bandwidthand computational power as streaming of unused tiles is avoided, but itcomes at the cost of increased computational complexity. However, mostoften, highly complex predictors do not have to be run for every renderof a virtual scene and low-complexity predictors may return inaccurateresults.

To solve these problems, there is a need for improved tile predictionthat also allows for better scaling of the computational load of thepredictions on the computer system.

The paper ‘Texture Virtualization for Terrain Rendering’ (D. Cornell,April 2012, pp. 1-10) relates to virtual texturing and presents anoverview of virtual texturing technology and virtual texturingpipelines. It discusses how a scene can be rendered to a need bufferwhich stores information on texture tiles in order to determine thevisible texture regions that need to be loaded in a streaming cache. Thepaper discusses various different stand-alone techniques which can beused by a predictor to determine these visible texture regions. Thepaper implies using a single predictor for determining said visibletexture regions, running in a static configuration for each frame.Hence, only stand-alone specialized techniques are presented. Thesestand-alone techniques are not shown to have any adaptive propertiesconcerning their activation or deactivation, frequency of execution,computational complexity or combination with other techniques.

The bachelor thesis ‘Virtual Texturing’ (A. Neu, April 2010) presents anoverview of virtual texturing technology and virtual texturingpipelines. As in the paper of Cornell, the thesis discusses how thescene can be rendered to a need buffer which stores information ontexture tiles in order to determine the visible texture regions thatneed to be loaded in a streaming cache. Various different stand-alonetechniques are discussed which can be used by a predictor to determinethe visible texture regions. Also here the use of a single predictor fordetermining the visible texture regions running in a staticconfiguration for each frame, is implied.

In US2008/147971 a predictive model is used to populate a cache in avideogame system. The predictive model takes as an input a sequence offile sections that have been requested for the associated videogame thusfar. The predictive model then returns the names or indicators of one ormore file sections that will likely be requested in the future alongwith a probability that those file sections will be requested. This canbe used to reduce the load times during the execution of the video game.

SUMMARY OF THE INVENTION

It is an object of embodiments of the present invention to provide for asolution that contributes in optimizing the streaming of objects to beused for rendering a virtual world of a computer graphics application.

The above objective is accomplished by the solution according to thepresent invention.

In a first aspect the invention relates to a prediction system fordetermining a set of subregions to be used for rendering a virtual worldof a computer graphics application, said subregions belonging tostreamable objects to be used for rendering said virtual world, saidstreamable objects each comprising a plurality of subregions. Theprediction system comprises

a plurality of predictor units arranged for receiving from a computergraphics application information on the virtual world and each arrangedfor obtaining a predicted set of subregions for rendering a virtualworld using streamable objects, each predicted set of subregions beingobtained by applying a different prediction scheme,

a streaming manager arranged for receiving the predicted sets ofsubregions, for deriving from the predicted sets a working set ofsubregions to be used for rendering and for outputting, based on theworking set of subregions, steering instructions concerning the set ofsubregions to be actually used.

The proposed solution indeed allows for optimized streaming whenrendering a virtual world of a computer graphics application. Thestreamable objects to be used for rendering are divided into a pluralityof subregions, usually referred to as tiles in the art. By performingtwo or more predictions of the set of subregions to be used forrendering, the set of subregions to be used for rendering can bedetermined more precisely, as well as the set of subregions that areused for rendering the future virtual scene of a virtual world. Multiplepredictors allow for more precise steering of a fine-grained streamingsystem for loading subregions to memory and allow for loading them intime such that the application has the subregions available upon firstuse, so improving the visual quality of the rendered scene. Multiplepredictors also allow for better scaling of the computational load ofthe prediction system on the computer system. The output of the multiplepredictors is combined and a working set proposal is derived. Thisworking set proposal is used to steer the streaming engine to manage thesubregions actually to be used and streamed.

In a preferred embodiment at least one streamable object is representedby subregions corresponding to a plurality of different level-of-detailversions of the at least one streamable object. Having several versionscorresponding to different resolutions allows selecting at any time theversion that yields the best render result in the given circumstancesand allows for faster fine-grained streaming of subregions of streamableobjects.

In one embodiment the streaming manager is arranged to exploitinformation on the accuracy of the plurality of predictions to optimizethe steering instructions. This information can be taken into accountwhen deciding on the set of subregions that will actually be used forrendering.

In another preferred embodiment the prediction system comprises apredictor controller arranged to select and control a subset of theplurality of predictor units and/or a timing scheme or a rate at whicheach predictor unit of said plurality works. The predictor controllercan control the prediction system, e.g. to trigger a predictor to run. Asystem with a predictor controller is clearly equivalent to a systemwithout such a controller, but where instead the prediction systemitself holds the controlling logic.

In one embodiment the subset and/or the timing scheme or the rate atwhich each predictor unit works, is determinable at run time. In anotherembodiment the subset and/or the timing scheme or the rate at which eachpredictor unit works, is determinable in accordance with availablesystem resources and/or based on the computational complexity of thepredictor units and/or based on the accuracy of results of the predictedsets of subregions and/or the delay at which the predicted sets becomeavailable. The proposed system indeed allows for multiple predictors tobe active at any given time in the application's runtime, where theproposed system chooses which subset of predictors is active at a giventime, and at which rate these predictors work. In such a way, predictioncan not only allow for more precise steering of the streaming system,but also continuously and dynamically adapt the prediction scheme to theapplication's requests, to the computer resources, to the complexity ofthe running schemes and to the accuracy of the predictor's output,throughout the application's runtime.

In a preferred embodiment the streamable object is a texture to bemapped to an object used for rendering a scene of the virtual world. Thesubregions, usually called tiles, are parts of the texture image.

In another embodiment streamable object is a geometry model used torender a scene of the virtual world.

In one embodiment one or more of the predicted sets of subregions areobtained at an own rate, i.e. the rate at which the predictor unityielding that predicted set works. For example, predictions that requireless computation can be performed more often than predictions with ahigh computational load.

In another embodiment one of the predicted sets is obtained bydetermining which streamable objects or which subregions are within agiven virtual distance in a virtual scene.

In one embodiment at least one of said prediction schemes comprisesrendering a scene from a virtual camera position and identifying aunique subregion identifier.

Advantageously, at least one of the prediction schemes is arranged forperforming a prediction whereby a virtual camera position in the virtualworld is exploited different from a main virtual camera position forrendering the application. This allows anticipating the tile visibility,as a future position of a virtual camera can be exploited.

Advantageously, at least one of the prediction schemes is arranged forperforming a prediction whereby the virtual world is rendered with nooccluding objects or wherein occluding objects are rendered with a levelof transparency.

In another embodiment the streaming manager is arranged for determiningthe set of subregions to be actually used by assigning a weight to thevarious subregions of the predicted sets. The weight may for exampletake into account the number of occurrences of a subregion in theplurality of predicted sets. This allows for prioritizing streaming ofsubregions.

In a preferred embodiment the streaming manager is arranged forselecting memory slots to be reused and for forwarding an indication ofthe selected memory slots in said steering instructions.

In one embodiment the streaming manager is arranged for outputtingsteering instructions by refining the output of one or more predictorsusing the output of at least one other predictor.

In a preferred embodiment the application is a video game softwareapplication.

In another aspect the invention relates to a method for determining aset of subregions to be used for rendering a virtual world of a computergraphics application, said subregions belonging to streamable objects tobe used when rendering that virtual world, said streamable objects eachcomprising a plurality of subregions. The method comprises

receiving from a computer graphics application information on a virtualworld,

obtaining a plurality of predicted sets of subregions for rendering avirtual world using streamable objects, each predicted set of subregionsbeing obtained by applying a different prediction scheme,

deriving from the predicted sets of subregions a working set ofsubregions to be used for rendering,

outputting, based on the working set of subregions, steeringinstructions concerning the set of subregions to be used.

In another aspect the invention relates to a program, executable on aprogrammable device containing instructions, which when executed,perform the method as described above.

In yet another aspect the invention relates to a video game consolecomprising a prediction system as previously described.

For purposes of summarizing the invention and the advantages achievedover the prior art, certain objects and advantages of the invention havebeen described herein above. Of course, it is to be understood that notnecessarily all such objects or advantages may be achieved in accordancewith any particular embodiment of the invention. Thus, for example,those skilled in the art will recognize that the invention may beembodied or carried out in a manner that achieves or optimizes oneadvantage or group of advantages as taught herein without necessarilyachieving other objects or advantages as may be taught or suggestedherein.

The above and other aspects of the invention will be apparent from andelucidated with reference to the embodiment(s) described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described further, by way of example, withreference to the accompanying drawings, wherein like reference numeralsrefer to like elements in the various figures.

FIG. 1 illustrates an embodiment of a data caching system according tothe invention.

FIG. 2 illustrates texture mapping.

FIG. 3 illustrates a 2D texture with multiple MW levels.

FIG. 4 illustrates a geometry model with multiple LOD levels, eachbroken into tiles.

FIG. 5 illustrates an embodiment of the proposed prediction system.

FIG. 6 illustrates another embodiment of a data caching system accordingto the invention.

FIG. 7 illustrates a timing scheme of a prediction system with twopredictors.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

The present invention will be described with respect to particularembodiments and with reference to certain drawings but the invention isnot limited thereto but only by the claims.

Furthermore, the terms first, second and the like in the description andin the claims, are used for distinguishing between similar elements andnot necessarily for describing a sequence, either temporally, spatially,in ranking or in any other manner. It is to be understood that the termsso used are interchangeable under appropriate circumstances and that theembodiments of the invention described herein are capable of operationin other sequences than described or illustrated herein.

It is to be noticed that the term “comprising”, used in the claims,should not be interpreted as being restricted to the means listedthereafter; it does not exclude other elements or steps. It is thus tobe interpreted as specifying the presence of the stated features,integers, steps or components as referred to, but does not preclude thepresence or addition of one or more other features, integers, steps orcomponents, or groups thereof. Thus, the scope of the expression “adevice comprising means A and B” should not be limited to devicesconsisting only of components A and B. It means that with respect to thepresent invention, the only relevant components of the device are A andB.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present invention. Thus, appearances of the phrases“in one embodiment” or “in an embodiment” in various places throughoutthis specification are not necessarily all referring to the sameembodiment, but may. Furthermore, the particular features, structures orcharacteristics may be combined in any suitable manner, as would beapparent to one of ordinary skill in the art from this disclosure, inone or more embodiments.

Similarly it should be appreciated that in the description of exemplaryembodiments of the invention, various features of the invention aresometimes grouped together in a single embodiment, figure, ordescription thereof for the purpose of streamlining the disclosure andaiding in the understanding of one or more of the various inventiveaspects. This method of disclosure, however, is not to be interpreted asreflecting an intention that the claimed invention requires morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive aspects lie in less than allfeatures of a single foregoing disclosed embodiment. Thus, the claimsfollowing the detailed description are hereby expressly incorporatedinto this detailed description, with each claim standing on its own as aseparate embodiment of this invention.

Furthermore, while some embodiments described herein include some butnot other features included in other embodiments, combinations offeatures of different embodiments are meant to be within the scope ofthe invention, and form different embodiments, as would be understood bythose in the art. For example, in the following claims, any of theclaimed embodiments can be used in any combination.

It should be noted that the use of particular terminology whendescribing certain features or aspects of the invention should not betaken to imply that the terminology is being re-defined herein to berestricted to include any specific characteristics of the features oraspects of the invention with which that terminology is associated.

In the description provided herein, numerous specific details are setforth. However, it is understood that embodiments of the invention maybe practiced without these specific details. In other instances,well-known methods, structures and techniques have not been shown indetail in order not to obscure an understanding of this description.

The present invention proposes a prediction system for optimizing astreaming system, wherein a plurality of subregions of streamableobjects is streamed to a memory of a computer graphics application to beused for rendering a virtual world. The proposed prediction systemreceives information on the virtual world from the computer graphicsapplication and predicts by means of a set of predictors, each applyinga different prediction scheme, a plurality of predicted set ofsubregions used for rendering said virtual world using said streamableobjects. A working set of subregions is derived from the resultingpredictions and a streaming manager component outputs steeringinstructions for the streaming engine concerning the set of subregionsto be actually used to render the application.

In the most preferred embodiments a solution is proposed for optimizinga streaming system which streams a plurality of subregions, possiblywith different levels-of-detail (LODs), to a memory in a renderingapplication. The subregions, which can be streamed independently, arereferred to as tiles. The presented approach steers the streaming of astreaming engine to the streaming cache by performing tile predictionswhich indicate which tiles are needed by the application forcomputational operations such as rendering the current or future virtualscene. The required set of tiles is called the working set. The systempreferably runs multiple predictors. The predictions can each beobtained with their own method/algorithm for predicting the current orfuture tile working set, their own accuracy of the results and their owncomputational complexity. The predictions may be performed at differentfrequency intervals in order to repeatedly and accurately predict thetile working set.

FIG. 1 is a high-level block diagram of an embodiment of a predictionsystem according to the present invention. The proposed system comprisesprediction means for performing two or more predictions and a streamingmanager and is connected to a computer graphics application, a streamingengine and a cache memory. The prediction system may be part of a gamingsystem, personal computer, workstation, server system, part of ahardware architecture or of a software architecture. The application maybe a rendering application rendering a 2D or 3D virtual world, or anyother application requiring streamable objects processing capabilities.Streaming cache memory includes processor readable memory to storestreamable objects used by the application to render. It may reside insystem memory, e.g., system RAM, or in GPU accessible memory, on adiscrete GPU with dedicated memory or in shared system memory accessibleby the GPU. The streaming engine loads, in response to requests itreceives from the application and the streaming manager, subregions ofstreamable objects, possibly with different levels of detail. It may bea dedicated component in hardware or a software module. The predictionsystem may be built in hardware (e.g., in a GPU), but it can also beimplemented as a module running in software on the system processor(s).A mixture of both options is possible as well.

The application may be a video game software application, a generalrendering application rendering a virtual scene. It may be a virtualreality application. It may render or process a single virtual scene ormultiple virtual scenes at once, for example, for 3D stereoscopicvision. The computing device whereon the application is run, may be avideo game console. Alternatively the application may be running on orthe computing device may be included in e.g. a cell phone, mobiledevice, embedded system, media console, laptop computer, desktopcomputer, server and/or data centre. The application may render and/ordisplay a virtual scene, or it may render a virtual scene and use therendering step output to carry out other calculations. The applicationmay render a two-dimensional or a three-dimensional scene. Theapplication may visualize a rendered scene directly to a screen, it maysend the results in the form of a video bit stream or it may process therendered scene to perform other calculations. The application may be arendering application designed to present and render the virtual world,the application may be designed to edit or author a virtual world, orboth present and edit the virtual world. For example, the applicationmay be an editor application for the virtual world of a video gamecapable of rendering the virtual world in an editor mode for editing thevirtual world, and render the virtual world in an end-user mode thatrenders the game as it would run, once its production is finalised andit is shipped to a gamer on, for example, a DVD-ROM. In this scenario,the invention can be used to predict the tile working set and embed tileworking set descriptors in the game content for the streaming engine touse once the game runs on the end user's device.

The application renders the scene using streamable objects (SOs) fromthe memory. A SO can be a texture which the application can map togeometry. A SO can also be geometry data, e.g., a mesh model built upwith polygons or points the application uses to render the scene. A SOcan also be more generalized data, such as animation key frames.Multiple level of detail (LOD) versions of a same SO may be present.Each LOD version of the SO describes the same render object, but with adifferent quality (or possibly a different amount of data to representthe SO). A plurality of LODs may be resident in memory to render thescene using the LOD version of the SO the application has selected(e.g., the LOD which yields the best sampling result). For example, alow-accuracy model of a mountain containing few geometrical data (lownumber of polygons for example) may be used when rendering a mountainfrom a high virtual distance in the virtual world, while a highlyaccurate model (containing many polygons) may be used when rendering themountain from close up. LODs may be streamed by the streaming engine.LODs may be generated by the application and derived from available LODsof a SO. In case a SO is composed of geometry data, different LODs formdifferent representations of the geometry model, each with a differentquality, for example, by means of more or less polygons to describe thegeometric model. In case a SO is a texture, LODs may include multiplerepresentations of an image, each with more or less pixels to describethe image. In case a SO is composed of model animation key frames, LODsmay include multiple representations of the animation, each with more orless key frames describing the transitions of the models in theanimation. A LOD may also represent multiple representations of a SOeach with a different level of quantization. For example, in case a SOis a texture, texture images may be stored with a smaller colourpallete.

A brief overview is now presented of texture mapping, such as can beperformed in conjunction with embodiments of the invention. In FIG. 2 atexture sample is applied to an object to result in a textured object.The texture sample is generally a graphics file that represents atexture one wishes to have mapped to the object. The object as shown inFIG. 2 is a three-dimensional object. However, the object can also be atwo-dimensional object, for example. By using a predetermined mannerknown in the prior art, the sample is applied to the object to result inthe textured object. The textured object has the same shape as theobject, but has on at least its visible surfaces the texture pattern.Sometimes a set of coordinates is set for each vertex of the shape,generally referred to as UV coordinates. These coordinates determinewhich region of the texture is mapped to the surface of the object. Thecoordinates typically have values between zero and one. For example, thefront face of the cube in the figure has four vertices, with UVcoordinates [0,0], [0,1], [1,0] and [1,1], designating that the entiretexture is mapped to the surface. For example, coordinates [0,0],[0,0.5], [0.5,0] and [0.5,0.5] designate that the first quarter of thetexture is mapped to the surface. Applying the texture to the object toresult in the textured object is referred to generally as texturemapping.

For example, a SO may be a texture. A texture can hold any type of twoor three-dimensional data, including but not limited to colour data,surface normal vectors and specular reflection data. The texture datamay comprise of multi-resolution texture pyramids, MW (multum-in-parvo)levels. At the lowest-level (level 0) of a pyramid is the fullresolution image. At the next higher level (level 1) of a pyramid is theimage at a lower resolution (for instance, 2×2 pixels or texels in thelowest level are averaged to form a single texel in the next highestlevel). At the next higher level is the image at an even lowerresolution. Moving towards the top of the pyramid, the image resolutionbecomes progressively lower. Much less data is needed to represent thelowest resolution image than the highest resolution image. Each MIPlevel represents a different level of detail.

A SO is split into a plurality of subregions, called tiles. Eachsubregion describes a part of the entire SO. These subregions may begeometric subregions of a SO, for example, 3D spatial regions of ageometry model or 2D spatial regions of a 2D texture. They may begeneral subchunks of memory of the memory buffer the SO describes. Ifthe SO holds multiple LODs, each LOD can be split into subregions(tiles). A tile with a certain LOD may be represented by one or moretiles on a different LOD. Subregions may comprise of an entire MIPlevel. In other words, a tile then corresponds to a single level of theLOD pyramid. Note that tiles do not need to have a uniform size orsubregion layout. The use of tiles allows for a more fine-grainedstreaming granularity. Many different ways to divide the SO intosubregions are available in the art. Tiles may hold compressed oruncompressed texture data. The data may be compressed using avariable-length compression format, such as ZIP or with a vectorquantization method, such as DXT1 or ASTC. Other compression schemes arepossible as the skilled person will readily appreciate. The data mayalso be uncompressed. The data of a tile may rely on the data of othertiles to represent a (part of a) SO. For example, a tile on level 3 mayhold data that is used to refine the data of a tile on level 4.

When an SO is a texture, the texture is broken up into tiles, i.e. insubregions of the texture image. If the texture contains multiple MIPlevels, each MIP level can be broken into tiles. FIG. 3 illustrates thisfor a two-dimensional texture. A tile on a certain MIP level may berepresented by multiple tiles on another MIP level. For example, in FIG.3 the tile on the highest level, i.e. the lowest-resolution level, isrepresented by four tiles on the next MW, which has a higher-resolution(lower level). Subregions may consist of an entire MIP level. In otherwords, a tile corresponds to a single level of the pyramid. It is notneeded that tiles all have the same size. Tiles may be 2D rectangularregions or 3D cubic regions. Two tiles may hold data from two differentMIP levels, in other words, they contain data with a differentlevel-of-detail (LOD). Each MIP level, from the highest-detailed texture(i.e., the lowest level on the pyramid) to the lowest-resolution texture(i.e., the highest level) of the texture may be split into tiles. Tilesmay comprise compressed or uncompressed texture data. The data may becompressed using a variable-length compression format, such as JPEG orJPEG XR, or with a vector quantization scheme, such as DXT1 or ASTC.Other compression techniques are possible. The data of a tile may relyon the data of other tiles to represent a (part of a) SO. For example, atexture tile on level 3 may hold data to refine the data of a tile onlevel 4, for example, by holding only high frequency image data that isadded to the lower-resolution image data of the tile on level 4.

For example, a texture of 1,024 by 1,024 pixels may hold 2D colourinformation. It has 11 MIP levels, i.e., the 1,024×1,024 image, level 0,a 512×512 level, i.e. level 1, a 256×256, this is level 2, 128×128,level 3, 64×64, level 4, 32×32, level 5, 16×16, level 6, 8×8, level 7,4×4, level 8, 2×2, level 9 and a 1×1 level, which is level 10. Thistexture can be split into 2D tiles with size 128 by 128 pixels. Level 0,the highest detailed level, holds 8×8, thus 64, tiles. Level 1 holds 16tiles, level 2 holds four tiles and level 3 one tile. Optionally alllevels can be omitted that are smaller than the tile size. In thisexample, all levels higher than 3 are omitted. One can also opt to putlevels smaller than the tile size in tiles. In this example, this meanslevel 4 with its image of 64 by 64 pixels would be put into a 128 by 128tile, just as level 5 up to 10.

When an SO is a geometry model, the model is broken up into tiles, i.e.in subregions of the model. If the model contains multiple LOD levels,each level is broken into tiles. FIG. 4 illustrates this for athree-dimensional model mesh built from vertices and triangles. A tileon a certain LOD level may be represented by multiple tiles on anotherLOD level. For example, in FIG. 4 the tile on the highest level, i.e.the lowest-resolution level, is represented by two tiles on the nextLOD, a higher-resolution, lower level. Subregions may consist of anentire LOD level. Tiles may hold 2D rectangular regions or 3D cubicregions. Two tiles may hold data from two different LOD levels. Each LODlevel, from the highest detailed model (i.e., the lowest level of thepyramid) to the lowest-resolution model (i.e., the highest level) may besplit into tiles. Tiles may hold compressed or uncompressed data. Thecompressed data may be compressed using a compression format such asZIP. Other compression techniques are possible.

For example, a geometry model as depicted in FIG. 4 may have four LODs,level 0, level 1, level 2 and level 3. Each LOD represents the samemodel of a person, but with a different quality. Each LOD is subdividedin subregions, tiles, each holding a number of the vertices thatdescribe the model. For example, level 2 is split into two tiles eachholding a half of the model. Level 1 is split into four tiles, eachholding a quarter of the model. Level 0 is split into six tiles, eachholding one sixth of the vertices of the model.

The streaming engine loads SO tiles into its cache memory in response torequests it receives. Streaming of SOs allows the use of many virtualobjects with accompanying SOs, for example, many virtual trees in avirtual world, with each their unique textures. There can be many morevirtual objects than what would fit into system or GPU memory. These SOtiles are loaded in a fast streaming cache memory that the applicationaccesses. The streaming engine may load the SO tiles from a slowerstorage medium, such as a hard disk. The storage system is typicallylarger than the cache memory, hence the streaming cache performing acaching function between larger but slower memory and the application.The requests the streaming engine receives originate from theapplication or from other operators, for example, a person who has set alist of tiles to be loaded when editing the virtual world. Suchoperators may fulfil the task of helping the streaming system streamingin the right tiles at the right moment.

Streaming many virtual objects allows for better management of bandwidthand computational power. The SOs of these virtual objects are streamedon a by-need basis. Streaming an unneeded SO wastes computer resources,as a SO takes up valuable memory and processing the SO to get it fromstorage to memory consumes processing cycles. Going further andstreaming SOs on a tile basis allows streaming on a by-need basis withmuch finer granularity and thus better use of bandwidth andcomputational power. Streaming SO tiles of a virtual object may involveamongst other things reading the SO from a storage device or network anddecompressing the read bit stream to a format comprehensible for therendering system. The streaming engine differentiates between LODs ofdifferent SOs. Multiple LODs of a SO may be requested and streamed tomemory and the set of LODs of a SO loaded in memory may continuouslychange.

For example, suppose a SO is a geometry model of a virtual mountain. Ifthe application renders a view on the virtual world of someone standingon a mountain, a SO containing the mountain's 3D model mesh for alow-level, highly detailed version will be requested by the applicationfrom the streaming engine. In case that same application renders a viewof the virtual world at a great distance of said mountain, a SOcontaining a 3D model mesh for a high level, a low-detailed version ofthe mountain, will be requested by the application from the streamingengine. In case of multiple mountains, the same SO may be requested andreside in memory with different LOD.

To allow for more fine-grained streaming, SO tiles are streamed by thestreaming engine. The application requests tiles instead of entire SOs.Streaming of SO tiles allows the use of virtual objects withaccompanying SOs, SOs that are much larger than what would fit into e.g.a system or GPU memory. It allows for better management of bandwidth andcomputational power. The tiles of the SOs may be streamed on a by-needbasis. Streaming a tile while it is not needed, is a waste of computerresources as a tile takes up valuable memory and processing the SOconsumes processing cycles. The streaming engine differentiates betweentiles with a different LOD. Multiple tiles with different LODs may berequested and streamed to memory and the set of LODs in memory of a tilemay continuously change.

As another example, suppose a SO is a texture of a 3D model of a virtualmountain. If the application renders a view on the virtual world ofsomeone standing on a mountain, tiles holding low-level LOD data, i.e.,highly detailed texture data for the mountain mesh will be requested bythe application to the streaming engine. Only tiles which hold texturedata of the front of the mountain may be requested. In case the sameapplication renders a view of the virtual world at a large distance ofsaid mountain, tiles containing high-level LOD texture data, i.e., lowdetailed texture data, for the mountain model mesh may be requested bythe application to the streaming engine. The tiles holding texture datafor the back of the mountain not visible to the virtual camera, may notbe streamed. In case of multiple mountains, the same tiles may berequested and reside in memory with different LODs.

The streaming engine may receive its requests directly from anapplication or from a rendering sub system or a different operator (e.g.a prediction system or another application). The requests may hold tileidentification information, possibly including LOD in case tiling iscombined with LOD. For example, the streaming engine may receiverequests for tiles for a texture SO based on a two-dimensional index anda LOD index. In another example, texture tiles are requested by using aone dimensional index and a LOD index.

A brief overview of a texture streaming system, such as can be used inconjunction with embodiments of the invention, is now presented. Thestreaming system, as shown in FIG. 1, stands between the application andthe streaming cache. It receives requests for a tile from theapplication or another operator. It has knowledge of which tiles arepresent in the cache memory and their position in memory. If a tile isrequested, the streaming system checks if the tile is present in cachememory. If it is present, i.e., a cache hit, the streaming systemimmediately returns the information to the application so theapplication can read the tile data from the streaming cache memory. Thestreaming system may return the address of the tile in memory and thetile LOD level. It may return the tile itself with its data. If therequested tile is not present in the cache memory, i.e., a cache miss,the streaming system streams the tile into the cache memory. Beforestreaming for the requested tile is finished, it may return the tilecache information for a LOD that does reside in cache memory. The tilewith the closest LOD can be returned. A ‘tile-not-found’ response may begiven. The set of all tiles requested through the streaming engine forrendering a scene is called the working set. Below a brief overview isgiven of the reasons for having a streaming cache and a predictionsystem. Those skilled in the art will appreciate that many streaming andtile-based streaming systems are described in the art and that thisinvention is not limited to those described in this text.

Streaming tiles from storage into memory can require a considerableamount of time. If the tile is streamed from a spinning disk to memory,the disk heads need to align, data needs to be read, tiles need to bedecompressed and copied to GPU texture memory. During this time the usermay be faced with a reduced quality of experience. The scene may berendered partially with no textures until the texture comes available,or it may just stop rendering until the texture comes available.

One way to improve the user experience if a tile is not present in thecache, is to render the scene with a tile with a lower LOD. However, theuser can be presented with detail that pops in the screen, i.e., theuser is confronted with a virtual object of which the rendering fidelitysuddenly changes as cache misses become cache hits once the streamingengine has streamed the tiles to the cache memory. This might presentitself as a very noticeable event in the rendering of the scene.

Clearly, it is preferable to have the tiles required to render a sceneavailable in cache memory and, preferably, to have them ready in thecache memory before or at the moment the application uses them for thefirst time. If the scene can be rendered solely with cache hits, therewill be no sudden popping of additional detail in the rendered resultsand the user can always experience the highest quality. By predictingthe working set of a virtual scene, i.e., the set of tiles to be used torender the scene, and/or by predicting the working set of future scenes,the cache can be loaded by the streaming system in such a way that cachemisses are minimized. Hence, the application does not have to wait untilthe tile is available for rendering, or does not have to resort torendering SOs with tiles of different LOD. The delay between a tilerequest and the moment a tile is available for the application caneffectively be hidden.

Furthermore, prediction allows better management of the streaming cachememory. When all cache slots, being a block of memory reserved for atile in the cache memory, are taken, certain cache slots are selected tobe reused, recycled and are flagged as available for the streamingengine to load new tiles in. By predicting the tile working set, tilesnot belonging to the working set can be identified, i.e., tiles that arenot necessary to render the scene. The cache slots for these tiles canbe offered to the streaming engine to be recycled, without any influenceon the rendering results.

A working set predictor, also named predictor, determines a predictionof a working set of a scene. It predicts which tiles will be requiredfor rendering or processing a virtual scene. Hence, it predicts thecontent or future content of the cache. By predicting the cache, thestreaming engine can be steered to fill the streaming cache in advanceand the tiles may be available in the cache the first time they areused.

There are many ways to determine a scene working set. Each method hasits own accuracy, typically coupled to computational complexity. Morecomplex algorithms most often need more computations. In other cases,algorithms have an increased delay, i.e., the time between the start ofthe calculations and the moment the calculation results are available issignificant. For example, running computations on a discrete graphicscard can cause a greater delay between calculation initiation and resultdelivery than performing the same on a CPU.

For a rendering application it is not always feasible to run highlycomplex prediction algorithms continuously. Low complexity algorithmsmay define the scene working set with sufficient accuracy for certainscenes or at certain times. Contrary to high-complex algorithms, theselow-complex algorithms can run continuously. By, for example, combiningthe output of low-complex continuously-running prediction algorithmswith that of higher-complexity, intermittently-running predictionalgorithms, visual fidelity may be increased enough to stop tiles withadditional detail popping in with the user noticing it. Runningpreferably multiple predictors each with their own method/algorithm andtheir own accuracy of the results and their own computationalcomplexity, each performed at different frequency intervals is the coreof this invention. Next, said predictors are discussed.

A predictor may run on the CPU and output its results to the systemmemory. A predictor may run on the GPU and output its results to GPUmemory or system memory. The following paragraphs list examples ofpredictors.

A SO may be a texture and a predictor may render a scene from thevirtual camera position of the application on the GPU using a pixelshader which outputs on a per-pixel basis the texture tile used,including its LOD level (in case of a texture, its MW level). Therendered output is then copied from the GPU memory to the system memorywhere the predictor loops over all rendered pixels and identifies theunique tile identifiers (including their LOD levels) present in thescreen. The cache predictor presents this unique list of tiles as itsoutput. Such a predictor can be best described as having a very highaccuracy and a high computational complexity, but because renderingasynchronously on the GPU introduces a delay typically in the order ofmilliseconds, such a predictor also introduces a high delay between thetime of initiation of the algorithm and the time of results.

The predictor as described above may render the scene from a virtualcamera position other than that of the main rendering view of theapplication. For example, the camera may be positioned at a point and adirection in the virtual world corresponding to a future or pastposition and direction of the virtual camera. A future camera position,for example, allows the system to predict the working set for a futureframe. The effect is that the cache system can anticipate tilevisibility.

The predictor as previously described may render the scene by renderingthe virtual objects in the scene as disclosed before, but may omit avirtual object responsible for occluding the virtual camera from thisrendering process. It may do this by deciding not to render the entirevirtual object or parts thereof or by rendering the virtual object usinga certain level of transparency resulting in occluded virtual objectsbecoming visible for the predictor and, hence, its results.

A SO may be a geometry model in the virtual scene and a predictor maytrace rays from the current or future virtual camera position andintersect them with placeholder geometry models in the virtual world forthe actual high-detailed models, running on the CPU. When a rayintersects with these placeholder geometric figures, the SO representedby the placeholder is identified and the distance between theintersection point and the camera is used to calculate the LOD level.The process is repeated for a number of rays originating from thevirtual camera. The list of unique SOs with all their tiles is presentedas the result of the predictor. Such a predictor is best described ashaving a medium accuracy, as it does not differentiate between tileswithin the same LOD for a SO. It also has a low delay as allcalculations are performed on the CPU synchronously and results areavailable almost immediately. The computational complexity depends onthe number of rays being traced each time the predictor runs.

A SO may be a texture and a predictor may request all tile referencesused throughout the entire virtual world from the application, a subsetof the virtual world or the loaded part of the virtual world, and listthese tile references as the working set.

A SO may be a texture and a predictor may request from the applicationall virtual objects within a determined virtual distance of theapplication's current, past and/or future virtual camera position. Thepredictor may inspect each object, determine the tiles the object canuse, and add those tiles to an internal list. The predictor outputs thislist as the working set.

The predictor may predict the working set by looking up a number ofvirtual objects and/or SOs, and (a subset of) their tiles in a datastructure. The data structure may be a quad tree, a BSP tree, k-d tree,other tree data structures, or other data structures including but notlimited to such as lists, linked lists, hash tables, dictionaries. Itmay store the entries in a spatial manner or allow spatial querying tofind the virtual objects, SOs or tiles. The data structure can beconstructed by the application at runtime. The data structure can begenerated in advance and loaded from memory, including storage memory,by the predictor.

The predictor may predict the working set or determine a number of tilesthat need to be streamed by looking up a precompiled list of tiles, orvirtual objects with their tiles, or SOs with their tiles. The predictormay be triggered into loading the tiles of the list by the applicationat certain moments in time. The predictor may be triggered to load thetiles of the list when the virtual camera is in the range of a virtualobject representing the list. The application may communicate the listto the predictor. The list can be constructed by the application atruntime. The list can be generated in advance and loaded from memory,including storage memory, by the predictor. For example, when anoperator of the rendering camera in the virtual scene enters a room, alist of all tiles in that room may be incorporated into the descriptionof the room in memory, the predictor may load the list and use itscontents as its prediction results.

The predictor may predict the working set by looking up all virtualobjects by shooting rays from a virtual camera position and finding theintersections of these rays with virtual objects. Once the virtualobjects intersecting the rays are found, i.e., all objects visible inthe scene, the objects' tiles are determined and added to an internallist. After processing all objects, this list is then communicated asthe working set.

FIG. 5 shows an embodiment of the proposed prediction system. In thepresented embodiment the prediction system comprises of a set ofpredictors, a working set manager and a streaming engine manager. Asexplained previously, the prediction system, together with theapplication, steers the streaming system into loading SO tiles into thestreaming cache memory. It does this by information it collects byrunning the predictors in the predictor set. These predictors mayrequest or share knowledge or information of the virtual scene renderedor processed by the application. Using this information, the predictorspredict the working set of tiles of the application for the current orfuture virtual scene. The predictors may run independently from theapplication. The predictors may also use information of the applicationon which tiles the application requests of the streaming engine.

In a preferred embodiment the system comprises a predictor controller tocontrol the prediction system, e.g. to trigger a predictor to run. FIG.6 illustrates this. A system with a predictor controller is clearlyequivalent to a system without such a controller, but where instead theprediction system itself holds the controlling logic, e.g. logic asdescribed with respect to the prediction system of FIG. 5. In oneembodiment this logic may even reside in the application, the streamingengine or any other component available in the system.

The predictor controller may request information from the application oruse information shared by the application, such as, but not limited to,information on the virtual rendering scene, rendering behaviour, etc.,in order to perform its controlling function. It may also useinformation on the computer system resources. For example, thecontroller may retrieve from the system the number of CPU cores and GPUsand determine the amount of predictors in its subset based on thesenumbers. For example, it may initiate a run of each predictor on adifferent CPU core or GPU and therefore make optimal use of theavailable resources.

At certain time intervals the prediction system runs a set of predictionalgorithms. The prediction algorithms may run asynchronously of theapplication without specific requests from the application to thestreaming system. The prediction algorithms may run synchronously, wherethe predictors may run each time the application requests a certain tileto the streaming system. The application may trigger a predictor to run.If present, a predictor controller may run or initiate a run at certainintervals of the set of prediction algorithms.

Each of the predictor algorithms calculates a proposal of the scene'sworking set of tiles and passes its results to the working set manager.The working set manager manages the different proposals of the sceneworking set and combines them to form a single scene working set. Thisworking set is communicated by the working set manager to the streamingengine manager. The streaming engine manager may detect changes to theworking set and manages the streaming engine accordingly. The streamingengine manager initiates steering instructions for the streaming engine,for example, by outputting loading requests for new tiles or bysignalling the streaming engine to free cache slots for tiles notpresent anymore in the working set. Note that the functionality of theworking set manager and the streaming engine manager can be combinedwithout departing from the scope of the invention. Also note that thepredictors can be integrated into the application without departing fromthe scope of the invention. The streaming engine manager can be part ofa streaming engine that accepts descriptions of the working set it needsto manage. This can be considered as a system sending a working setloading operation.

The predictor set comprises a number of predictors. The predictor setmay contain predictors for different sorts of SOs, for example, onepredictor for textures and one for point clouds. Each predictor may runor initiate its calculations continuously at a certain frequencyaccording to a timing scheme. The predictors may run synchronously orasynchronously to the application. The application may trigger apredictor to run. The timing scheme may describe the frequency at whichthe predictor is run in Hertz. For example, a high accurate predictor isset to run at a frequency of 20 Hz, i.e., every 50 milliseconds. For anapplication rendering a virtual world at 60 Hz, this corresponds torunning the predictor every three rendering frames. Alternatively, thetiming scheme indicates at which moments a predictor is run byspecifying the number of rendering frames between each initiation. Forexample, the timing scheme specifies a number of three frames betweeneach initiation. If the application renders the scene at 30 Hz, thismeans the predictor will run at 10 Hz. The predictor frequency may besynchronized to the rendering frequency. For example, a predictor mayrun its calculations in the same thread that is responsible for issuingthe rendering commands to the rendering API. This thread may initiate arun each time the thread starts rendering a frame. Instead of theprediction system or the application, a predictor controller may defineor impose the timing scheme.

FIG. 7 illustrates an example. Where a SO is a texture, the predictorset may contain two predictors. One of the predictors is a highlyaccurate, highly complex predictor which renders the scene as describedabove. The other predictor is a lower accurate, low complex predictorwhich lists all tiles within a certain virtual distance of the virtualcamera. The first predictor is run every five rendering frames, whilethe second predictor runs every rendering frame. The figure shows howthe first predictor takes up a significant amount of time until theresults are presented to the streaming engine manager, namely, after 2frames. The second predictor, however, presents its results within thesame rendering frame to the streaming engine manager. Note that in theexample the first predictor algorithm is not run for frames 4 and 5. Theresults may take additional time until they become available in thesystem memory. For example, rendering an image on the GPU and readingback the resulting image takes a similar delay of 2 to 3 frames. Asillustrated by this example, at specific times there may be only asingle predictor active (e.g., frames 4-5). However, over the course ofthe application's running time, multiple predictors, two in thisexample, are used.

In one embodiment, if a predictor is set to be initiated while aprevious instance of that predictor is still running, the predictor isnot scheduled to run. For example, if a predictor takes 10 ms to run andit is scheduled to run at 120 Hz, then the predictor will run at anactual frequency of 100 Hz. In another embodiment, if a predictor is setto be initiated while a previous instance of that predictor is donerunning, but its results are still being copied to memory, the newinstance may be initiated. For example, the results of a predictorprocess running on the GPU can be copied asynchronously to systemmemory, while a new instance of that same predictor is initiated. Inanother embodiment, if a predictor is set to be initiated while aprevious instance of the same predictor is still running, the predictormay still be initiated for a new run.

In one embodiment the initiation of the predictor may be triggered bythe application or another operator. The predictor may run continuouslyor may run a certain number of times. In an embodiment of the systemwith a predictor controller, it may be the predictor controller thattriggers the initiation of the predictor. The predictor controller mayreceive a signal from the application to initiate a predictor.

In one embodiment all predictors or a subset of predictors can run atthe same frequency. They may present their results at different times,as some predictors may need a longer time to calculate than others. Thepredictor controller, if present, or else the prediction system itselfmay impose this frequency or rate on the predictors.

It may be that only a subset of all available predictors is run. Thissubset can be determined at run time. For example, the number of activepredictors is chosen in accordance with the number of CPU cores in asystem. The subset can be predetermined by the system operator, forexample, by means of a user-set constant in the application. Aspredictors may run at different frequencies, it is possible that at acertain time in the application's lifetime/runtime only one predictor orno predictors at all is running, but that over the application'slifetime, multiple predictors are active. If the prediction systemcomprises a predictor controller, the latter may hold the logic todetermine, possibly at run time, which subset of predictors to run atwhich time interval. During the application's lifetime, the predictorcontroller or the prediction system may continuously determine thissubset and at which time it runs, and may continuously change the subsetand the timing scheme. For example, the prediction system continuouslyqueries the operating system for the computational load on each CPU coreand changes the active subset of predictors accordingly. For example,when the prediction system finds out the system is under heavy load, itchanges the subset of active predictors to a subset that exhibits alower computational complexity or in another example, it changes thesubset to a subset that uses more GPU resources and, hence, alleviatesthe CPU cores.

The predictor controller, if present, or else the prediction systemitself may have knowledge of the accuracy of the outcome of eachpredictor and/or the computational complexity of the predictor, andadjust the subset of predictors to run and/or the timing schemeaccordingly. For example, the controller may adjust the run frequency ofa predictor by lowering it, knowing that a predictor is highly accurate,but highly complex. The predictor controller or prediction system maymeasure the predictor accuracy and/or complexity, or may query theapplication, streaming manager, streaming system, computer system or anyother system to retrieve this information, at run time or based onpredetermined values. Based on this measurement, it can adjust thesubset of predictors to run and/or adjust the timing scheme. It may dothis continuously throughout the run time of the application.

A system running one or more predictors is clearly equivalent to asystem running a single predictor performing the prediction algorithmsof said one or more predictors and combining the output to a single setof predicted subregions. Such a system can be considered equivalent tocombining the predictors, streaming manager and predictor controllerinto one logical unit.

Each predictor run by the system presents its results to the tileworking set manager. The tile working set manager combines multipleworking set proposals received from the set of predictors, into a singleworking set. This working set is communicated to the streaming enginemanager, which steers the streaming engine accordingly. For example, thestreaming engine manager may output steering instructions to load aworking set, it may output steering instructions to unload specificsubregions, to remove subregions from the streaming cache, etc.

A tile working set is a list of tile references (including those ofdifferent LOD) the cache system should have in memory for best renderingresults. The tile references may comprise a tile index and LODindication.

The working set manager may have knowledge of the accuracy of eachpredictor in the predictor set. Therefore, it can distinguish betweeninaccurate and accurate working set descriptions. The working setmanager can hold for each predictor a range of valid LOD levels. Theworking set manager loops over all tile references in the proposedworking set and discards tile references with a LOD outside of thisrange. The remaining tile references are added to the refined workingset. In one embodiment, the working set manager does not have knowledgeof the accuracy of each predictor, but each predictor only outputsresults for the tiles it can predict with high accuracy.

For example, in case the SO is a texture and two predictors are active:a highly accurate predictor rendering the scene and a low-accuracypredictor which iterates all tiles within a given virtual range. Theworking set manager can hold a range of LOD level [0, 1] for the firstpredictor and a range of LOD levels of [2, 10] for the second predictor.For the first predictor's results, the working set manager discards alltile references with LOD level higher than 1. In other words, only thehighest-detailed two MW levels remain in the working set. For the secondpredictor's results, the working set manager discards all tilereferences with LOD level smaller than 2. In other words, only thelowest-detailed MIP levels (all but the two highest-detailed MIPs)remain. The working set manager combines these two working sets byconcatenating the two working sets. In other words, the first predictoris used to find the tiles visible in the screen with the highest detail.The second predictor is used to find the lower resolution tiles thatwill be visible in future frames when the first predictor is notrunning.

In one embodiment the working set manager assigns weights to each tilereference in the working set proposal presented by a predictor. Theseweights may be in relation to the predictor accuracy.

For example, in case the SO is a geometry model, the cache system holdstwo predictors, one performing the ray tracing algorithm as discussedearlier and the other iterating all SO tiles in a given distance fromthe current, past or future virtual camera position, also as discussedabove. The working set receives the results from the first predictor andassigns a weight to each of the tile references in the working set. Forexample, the tile description resulting from the second predictor canhave a smaller weight than that of the first predictor, therebyinstructing the working set manager to prioritize the second predictoroutput. The working set manager can choose to drop tile references basedon their weights. Another option is not to drop tiles, but passing theweights through to the streaming engine manager which can then performits cache replacement strategy based on the weight. Also, a framecounter, increasing each time a new frame is rendered by theapplication, can be incorporated into the weight. The current time couldbe incorporated. These approaches may give additional weight to thenewest results.

In one embodiment the working set manager may combine the predicted setsof the predictors into a single working set by optimizing or refiningthe output of one or more predictors by analysing the output of otherpredictors. For example, a predictor may output a set of subregionscorresponding to every visible subregion using a virtual camerarendering with a wider field of view (FOV) than normal. Anotherpredictor outputs a set of subregions corresponding to a narrower FOV.The working set manager may receive both sets, refine the setcorresponding to the wide FOV by removing subregions occurring in theset with the narrow FOV, and output subregion loading instructions tothe streaming engine for the resulting set of subregions.

In another embodiment the working set manager may receive a set ofpredicted subregions from a predictor, determine the working set as thisset of subregions and let the streaming manager prioritize thecorresponding steering instructions for the streaming engine by lookingat the output of another predictor. In such a system multiple predictorsare active, but the output of the predictors determines the priority orthe order of the steering instructions the streaming manager outputs.For example, the output of all predictors can easily be aggregated andsent to the streaming manager, but the order in which the streamingmanager outputs the steering instructions can be determined by theoutput of the individual predictors. For example, with two predictors asdiscussed in the example in the previous paragraph, the working setmanager receives the output of a predictor with a wide FOV and thestreaming manager outputs steering instructions for each subregion inthe output set of predicted subregions of said predictor. However, theorder in which the steering instructions are outputted is given by theworking set manager by analysing the output of the second predictor witha narrow FOV. Specifically, subregions present in the set correspondingto the wide FOV but not the one corresponding to the narrow FOV aregiven a higher priority.

In one embodiment the working set manager may combine working sets itreceives from the predictors implicitly by relying on the underlying oravailable systems or application. For example, the working set managermay aggregate the working sets it receives and send them to thestreaming manager. The streaming manager steers the streaming engine tostream every subregion in the working set and relies on the underlyingoperating system to eliminate duplicate requests to load a file frommemory.

After the working set manager has combined the proposed working sets ofthe predictors into a single working set, the resulting working set ispresented to the streaming engine manager. The streaming engine managersteers, based on the working set it receives, the streaming engine instreaming the required tiles not available in the cache. When itreceives a working set, it loops over all tile references. Tiles alreadyresident in the cache memory remain resident and new tiles are signalledto be streamed to the cache. The streaming engine manager may approachthe cache using a number of strategies. For example, it may use aleast-recently-used (LRU) strategy, where tiles that were theleast-recently used, are replaced in the working set. Various workingset combination strategies are applicable.

In one embodiment the streaming engine manager may just present thestreaming engine steering instructions to load the resulting working setand rely on the streaming engine to stream the working set. In anotherembodiment the streaming engine manager may rely on functionality of thestreaming engine to help manage or hint the use of, amongst others, thestreaming cache and streaming priority. In another embodiment thestreaming engine manager may compare the working set of subregions itreceived from the working set manager or the predicted sets ofsubregions from the predictors with information on subregions kept inmemory, for determining steering instructions to steer the streamingengine. In such a way, for example, the streaming engine manager cansteer the streaming engine manager into streaming more precisely.

The streaming engine manager can flag cache slots which are not used ina period of time as available for holding newly streamed tiles. Inanother embodiment the streaming engine manager receives the weights ofthe tiles from the working set manager, and replaces tiles according totheir weights. For example, tiles slots holding tiles with a smallerweight make room for tiles with a larger weight. In another embodiment aframe counter, increasing each time a new frame is rendered by theapplication is incorporated into the weight. Also the current time canbe incorporated. These two approaches may give additional weight to thenewest results and instruct the streaming engine manager to prioritizethe newest results. Several cache replacement strategies are applicable,not limited to the examples given here.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, such illustration and descriptionare to be considered illustrative or exemplary and not restrictive. Theforegoing description details certain embodiments of the invention. Itwill be appreciated, however, that no matter how detailed the foregoingappears in text, the invention may be practiced in many ways. Theinvention is not limited to the disclosed embodiments.

Other variations to the disclosed embodiments can be understood andeffected by those skilled in the art in practicing the claimedinvention, from a study of the drawings, the disclosure and the appendedclaims. In the claims, the word “comprising” does not exclude otherelements or steps, and the indefinite article “a” or “an” does notexclude a plurality. A single processor or other unit may fulfil thefunctions of several items recited in the claims. The mere fact thatcertain measures are recited in mutually different dependent claims doesnot indicate that a combination of these measures cannot be used toadvantage. A computer program may be stored/distributed on a suitablemedium, such as an optical storage medium or a solid-state mediumsupplied together with or as part of other hardware, but may also bedistributed in other forms, such as via the Internet or other wired orwireless telecommunication systems. Any reference signs in the claimsshould not be construed as limiting the scope.

The invention claimed is:
 1. A system comprising: one or more computerprocessors; one or more computer memories; and a set of instructionsincorporated into the one or more computer memories, the set ofinstructions configuring the one or more computer processors to performoperations for facilitating rendering of a virtual world of a computergraphics application, the operations comprising: receiving datadescribing the virtual world from the computer graphics application;analyzing the received data using a plurality of predictors, theanalyzing including applying a different prediction scheme according toeach predictor of the plurality of predictors to identify a predictedset of subregions belonging to streamable objects to be used forrendering the virtual world; combining the predicted sets of subregionsinto a working set of subregions to be used for the rendering, thecombining based on an analysis of output from each predictor;determining priorities of steering instructions for loading subregionsof the working set into a streaming cache memory to be used for therendering, the determining of the priorities based on an analysis ofoutput from each predictor; and outputting the steering instructions toa streaming engine to perform the loading of the subregions in order ofthe priorities.
 2. The system as in claim 1, wherein at least onestreamable object is represented by subregions corresponding to aplurality of different level-of-detail versions of the at least onestreamable object.
 3. The system as in claim 1, the operations furtherincluding applying a timing scheme for the application of a predictionscheme by one or more predictors of the plurality of predictors.
 4. Thesystem as in claim 3, wherein the timing scheme is determined at a runtime of the application.
 5. The system as in claim 3, wherein the timingscheme is determined in accordance with available system resources. 6.The system as in claim 3, wherein the timing scheme is determined basedon a computational complexity of a prediction scheme applied by the oneor more predictors.
 7. The system as in claim 3, wherein the timingscheme is determined based on an accuracy of results of the predictedsets of subregions and/or a delay at which the predicted sets becomeavailable.
 8. The system as in claim 1, wherein each prediction schemehas a different measure of accuracy of the output.
 9. The system as inclaim 1, wherein one of the predicted sets is generated by determiningwhich streamable objects or which subregions are within a predeterminedvirtual distance from a position in the virtual world.
 10. The system asin claim 1, wherein at least one of the prediction schemes comprisesrendering a scene from a virtual camera position in the virtual worldand identifying a set of unique subregion identifiers associated withstreamable objects visible by a virtual camera at the virtual cameraposition, and outputting the set of unique subregion identifiers. 11.The system as in claim 1, wherein at least one of the prediction schemesis configured for performing a first prediction associated with a mainvirtual camera position in the virtual world, and a second predictionassociated with a second virtual camera position, the second virtualcamera position being associated with a future main virtual cameraposition.
 12. The system as in claim 1, wherein at least one of theprediction schemes is configured for performing a prediction wherebystreamable objects in the virtual world are rendered from a point ofview with no occluding objects or wherein occluding objects are renderedwith a level of transparency.
 13. The system as in claim 1, theoperations further including assigning a weight to the varioussubregions of the predicted sets, the weight determining a priority forinclusion in the working set and a priority within the steeringinstructions.
 14. The system as in claim 13, wherein the weight takesinto account a number of occurrences of a subregion in the plurality ofpredicted sets.
 15. The system as in claim 1, the operations furtherincluding selecting memory slots to be reused and forwarding anindication of the selected memory slots in the instructions.
 16. Thesystem as in claim 1, the operations further including: modifying anoutput of one or more predictors using the output of at least one otherpredictor; and performing at least one of adjusting the combining of thepredicted sets of subregions based on an analysis of the modified outputand adjusting the determining of priorities based on an analysis of themodified output.
 17. The system of claim 1, wherein the combining of thepredicted sets of subregions into the working set of subregions to beused for the rendering is based on an analysis of a measure of outputaccuracy from each predictor.
 18. The system of claim 1, wherein thedetermining of the priorities of steering instructions is based on ananalysis of a measure of output accuracy from each predictor.
 19. Anon-transitory computer-readable storage medium storing a set ofinstructions that, when executed by one or more processors, causes theone or more processors to perform operations for facilitating renderingof a virtual world of a computer graphics application, the operationscomprising: receiving data describing the virtual world from thecomputer graphics application; analyzing the received data using aplurality of predictors, the analyzing including applying a differentprediction scheme according to each predictor of the plurality ofpredictors to identify a predicted set of subregions belonging tostreamable objects to be used for rendering the virtual world; combiningthe predicted sets of subregions into a working set of subregions to beused for the rendering, the combining based on an analysis of outputfrom each predictor; determining priorities of steering instructions forloading subregions of the working set into a streaming cache memory tobe used for the rendering, the determining of the priorities based on ananalysis of output from each predictor; and outputting the steeringinstructions to a streaming engine to perform the loading of thesubregions in order of the priorities.
 20. A method comprising:performing, using one or more computer processors, operations forfacilitating rendering of a virtual world of a computer graphicsapplication, the operations comprising: receiving data describing thevirtual world from the computer graphics application; analyzing thereceived data using a plurality of predictors, the analyzing includingapplying a different prediction scheme according to each predictor ofthe plurality of predictors to identify a predicted set of subregionsbelonging to streamable objects to be used for rendering the virtualworld; combining the predicted sets of subregions into a working set ofsubregions to be used for the rendering, the combining based on ananalysis of output from each predictor; determining priorities ofsteering instructions for loading subregions of the working set into astreaming cache memory to be used for the rendering, the determining ofthe priorities based on an analysis of output from each predictor; andoutputting the steering instructions to a streaming engine to performthe loading of the subregions in order of the priorities.